Sample function properties of two-parameter Markov processes

Bounded Parameter Markov Decision Processes Bounded Parameter Markov Decision Processes

In this paper, we introduce the notion of a bounded parameter Markov decision process as a generalization of the traditional exact MDP. A bounded parameter MDP is a set of exact MDPs speciied by giving upper and lower bounds on transition probabilities and rewards (all the MDPs in the set share the same state and action space). Bounded parameter MDPs can be used to represent variation or uncert...

Bounded Parameter Markov Decision Processes

In this paper, we introduce the notion of a bounded-parameter Markov decision process (BMDP) as a generalization of the familiar exact MDP. A bounded-parameter MDP is a set of exact MDPs specified by giving upper and lower bounds on transition probabilities and rewards (all the MDPs in the set share the same state and action space). BMDPs form an efficiently solvable special case of the already...

Local times and Sample Function Properties of Stationary Gaussian Processes

Ergodic properties of Markov processes

Warning. If F and G are σ-algebras, H = F ∪G is not necessarily a σ-algebra unless of course G ⊂ F or vice versa. Indeed, even if {Fn} is a countable collection of σ-algebras satisfying Fn ⊂ Fn+1 for each n = 1, 2, . . ., the union ⋃∞ n=1Fn is not necessarily a σ-algebra. For example, let Ω = N and let Fn be the smallest σ-algebra containing {1}, {2}, . . . , {n}. (This is the power set of {1, ...

Ergodic properties of Markov processes

In these notes we discuss Markov processes, in particular stochastic differential equations (SDE) and develop some tools to analyze their long-time behavior. There are several ways to analyze such properties, and our point of view will be to use systematically Liapunov functions which allow a nice characterization of the ergodic properties. In this we follow, at least in spirit, the excellent b...